feat: move GitLab from EKS to EC2 docker-compose#755
Merged
allamand merged 232 commits intoJul 7, 2026
Merged
Conversation
695eafe to
2ed034d
Compare
- Disable GitLab ArgoCD addon (enabled-addons, hub-config, platform.yaml registry) - Remove Keycloak SSO client for GitLab - Remove gitlab-nlb/gitlab-distribution Taskfile.cloudfront.yaml tasks - Add gitlab:init-ec2 task: wait for CE readiness, create root token, user1, repos - Replace k8s Job wait in clone-repos with CF readiness poll - GITLAB_DOMAIN_INT uses EC2 private IP for in-cluster ArgoCD git access - Remove git_token from seed-secrets (seeded by CDK at deploy time) Refs #754
f5ad84b to
ea0b1b1
Compare
Added: urls, hub:set-overlay-repo, hub:restart-langfuse, hub:wait-for-full-sync, secrets-manager:seed-secrets, secrets-manager:seed-observability, hub:create-mgmt-roles, hub:restart-identity-pods, hub:update, spokes:enable-crossplane/kro, spokes:create-capabilities, spokes:disable-crossplane/kro/all, spokes:seed-provider-identity Removed: idc:setup from install task (moved to workshop:Taskfile.yaml)
- Add parallel install phases (phase1/phase2) to kind-kro-ack install task - Move gitlab:init-ec2 + gitlab:clone-repos to workshop/Taskfile.yaml - crossplane-system refs in copied tasks left intentional (Crossplane addon on hub EKS)
…es managed declaratively by RGD
- Restore argocd login block in ssm-setup-ide-logs.sh - Restore progressive-app image_name/service name (rollout-demo → progressive-app)
…xists, project exists)
…lex hash/validation
RGDs ekscluster.kro.run and eksclusterwithvpc.kro.run were Inactive because: 1. ACK SecretsManager controller was missing (rg-eks.yaml references secretsmanager.services.k8s.aws/v1alpha1) 2. ESO was not installed on the bootstrap kind cluster (rg-eks.yaml references external-secrets.io/v1) Fixes: - Add ACK_SECRETSMANAGER_VERSION=1.3.1 and install it in ack:install - Add kro:install-eso-bootstrap task (idempotent) before kro:apply-rgds
…dd argoCdCapabilityRoleArn - Remove spec.argocdCapability block (not in EksclusterWithVpc RGD schema, causes strict decode error) - Add missing argoCdCapabilityRoleArn field (required by RGD) - Remove now-unused IDC sed substitution lines
… proper user+namespace creation
…lity Port from feature/platform-cluster-kro-ack: - rg-eks.yaml: add argocdCapability schema object, replace accessEntryArgoCdCapability with conditional argocdCapabilityRole + argocdCapability (EKS Capability with IDC) + argocdCapabilityAccessEntry (all guarded by includeWhen enabled==true) - rg-eks-vpc.yaml: add argocdCapability schema object, pass through to nested EksCluster - hub:claim: restore argocdCapability block with IDC sed substitutions
…otect after creation
…CRD with removed property
Deletes eksclusters.kro.run and eksclusterwithvpcs.kro.run CRDs on the bootstrap kind cluster. Needed when RGD schema changes remove fields (KRO breaking-change protection blocks the update otherwise). Run: task kind-kro-ack:kro:reset-crds, then task install.
Hardcoded 10.0.x.0/24 subnets break when vpcCidr is not 10.0.0.0/16. Add HUB_VPC_PREFIX var (first two octets of HUB_VPC_CIDR) and use it in hub:claim subnet CIDRs.
…xist When the EKS ArgoCD Capability is used, ArgoCD CRDs are already present on the hub cluster. Helm install then fails with 'CRD already exists'. Add a second status check: skip if applications.argoproj.io CRD exists.
… restart credential-less pods (#8) #7: ArgoCD's fleet-config-repo (repository) and gitlab-user-repo-creds (repo-creds) secrets were seeded imperatively once and never refreshed, so a rotated peeks-hub/secrets.git_token left ArgoCD unable to pull GitLab repos (manual patch per event). New argocd-gitlab-repo-creds chart provisions both as ExternalSecrets sourcing git_token (refreshInterval 5m, creationPolicy Owner). Hub-only (environment=control-plane) and served from the GitHub defaults path so ArgoCD does not need the very GitLab creds it provisions (breaks the circular dep). #8: KubeVela apps whose service-account (PodIdentityAssociation) and workload components render in parallel can start pods before the PIA is active; EKS injects pod-identity creds only at pod creation, so those pods fail CredentialsNotLoaded until restarted. New podidentity-pod-restart CronJob (gated enable_kubevela) deletes pods that lack AWS_CONTAINER_CREDENTIALS_FULL_URI but whose SA has a PodIdentityAssociation (Crossplane upbound or ACK), so they respawn mutated. Implemented as a standalone self-healing addon rather than editing the KubeVela ComponentDefinition CUE (lower blast radius, plain-YAML, validatable).
…Kargo steps From the crossplane KubeVela-path validation: #5 (healthcheck 404): set healthcheckPath: /collection/FRONT_PAGE on the Rust app path-based-ingress trait — the app only serves 2xx under /collection/*, so the ALB target health check must not use the default "/". #6 (KubeVela rollout not triggered): the CI image-update step only rewrote image: — KubeVela only re-renders the Rollout when app.oam.dev/publishVersion changes. CI now also bumps publishVersion to the image tag (java + non-java branches; no-op for manifests without the annotation). #7 (Kargo 'Stage defines no promotion steps'): inline the promotion steps into each Stage's promotionTemplate instead of a PromotionTask reference, which some Kargo versions fail to resolve at admission. promotiontask.yaml retained for reference (no longer referenced). #8 (Kargo dotted key 'key path not found'): address the KubeVela app.oam.dev/publishVersion annotation with bracket notation metadata.annotations["app.oam.dev/publishVersion"] instead of dot-escaping. Kept continueOnError (kro-variant manifests lack the annotation). NOTE: #7/#8 are Kargo-version-sensitive and were validated only at YAML level; runtime verification pending on the live Kargo (#8 stays non-blocking via continueOnError regardless).
…rRole The Ray RGD creates PV+PVC for S3 model storage (Mountpoint CSI). Without these resources in the ClusterRole, kro gets 'forbidden' and the RayService stays in ERROR.
- Use inline steps instead of task-ref (Kargo v1.7.5 webhook rejects task-ref with 'Stage defines no promotion steps') - Remove publishVersion annotation update (yaml-update cannot address keys with dots — neither bracket notation nor dot-escaping works) - Enable auto-promotion for both dev and prod stages (workaround for webhook bug blocking manually-created Promotions) - Mark promotiontask.yaml as reference-only (no longer applied by deploy-kargo.sh) Validated end-to-end: CI push → Warehouse detects image → dev auto-promotes → prod auto-promotes → git repo updated with new image tag.
… ALB host wildcard - kro-ack cluster-mgmt trust: add peeks-spoke-*-ack-capability-role (blocker: spoke ACK AccessDenied sts:TagSession); mirror crossplane - both providers eks-access-management: PassRole for *-microservice-* app roles (blocker: EKS PodIdentityAssociation on rust-microservice-dev-role) - appmod-service RGD: correct default host wildcard *.us-west-2.elb.amazonaws.com (was elb/us-west-2 labels swapped -> ALB 404)
Replace the exposure_mode==cloudfront toggle in the platform ingress
templates (keycloak, grafana, jupyterhub, argo-workflows, kubeflow,
backstage) with an explicit `insecure` flag:
insecure=true -> HTTP:80 listener, no cert, no host rule
(consumer terminates TLS upstream — CloudFront, etc.)
insecure=false -> HTTPS:443 + ssl-redirect + host rule (ACM on the ALB)
The flag is decoupled from CloudFront: any consumer that fronts the ALB
with a TLS-terminating edge sets insecure=true; customers with their own
domain + ACM leave it false (the secure default).
Backward-compatible during the transition: `insecure` falls back to the
legacy exposure_mode=="cloudfront" check when unset, so current installs
keep rendering identically until the provider/config layers are refactored.
Verified with helm template: all six render HTTP:80 (insecure) and
HTTPS:443+host (secure).
…ontract Additive step toward the generic exposure model: - `insecure` (bool, default false): true = HTTP-only ingress, consumer terminates TLS upstream; false = ALB HTTPS with ACM. - `certificateArn` (optional): explicit ACM ARN for the HTTPS listener; empty = LBC auto-discovers by host. - `domain` reworded: consumer-provisioned ingress hostname; the platform never creates it. cloudfront.* fields left in place for now — they're removed together with the provider CloudFront-provisioning code in a later commit to avoid breaking the current install path mid-refactor.
…kshop/ folder
Workshop is a CONSUMER of the platform, not a cluster provider:
- move cluster-providers/workshop/{Taskfile,README} -> workshop/
- move solutions/ (Module 1 answer-key RGDs, peeks-specific) -> workshop/solutions/
- update the root Taskfile 'workshop:' include to ./workshop
No behavior change: 'task workshop:*' resolves identically (verified with
task --list). This establishes workshop/ as the single home for
workshop-specific logic so the rest of the repo can be the generic solution.
…n from config NEEDS LIVE-INSTALL VALIDATION. The generic provider no longer creates a CloudFront distribution or the internal platform ALB. Instead: - domain is consumer-provided: ingress_domain_name := config .domain (always), in hub:seed and urls. Removed the private/cloudfront-domain reads + AWS distribution lookups. - exposure toggle is driven by the new `insecure` flag: INSECURE := config .insecure; EXPOSURE_MODE derived (insecure->cloudfront/HTTP, else domain/ HTTPS) so the existing cluster-secret annotation chain the addon charts consume keeps working. Seed metadata now also carries insecure + certificate_arn. - removed the cloudfront: include, the cloudfront:setup-exposure/sync-domain calls in install, install:phase1-cloudfront (-> phase1-identity, restart identity pods only), and the hub:update-ingress-domain task. common/Taskfile.cloudfront.yaml is left in place for now — kind-kro-ack still includes it; it is removed together with the kind-kro-ack decoupling next. Verified only: YAML parses, task lists 40 kind-crossplane tasks, install summarizes, no dangling cloudfront: refs. NOT install-tested.
…F taskfile NEEDS LIVE-INSTALL VALIDATION. Mirror the kind-crossplane change on the kro-ack provider and remove the now-unused shared task file: - delete cluster-providers/common/Taskfile.cloudfront.yaml (no longer included by either provider). - kind-kro-ack: drop the cloudfront: include + cloudfront:setup-exposure/ sync-domain calls; source ingress_domain_name from config .domain; derive exposure_mode from the new `insecure` flag; align cert to top-level .certificateArn; seed metadata carries insecure + certificate_arn. Both providers now assume the consumer provisions the domain + TLS edge; the platform only consumes `domain` + `insecure`. Verified only: both Taskfiles parse (yq), root 'task --list' resolves all includes, kind-crossplane=40 / kind-kro-ack=47 tasks. NOT install-tested. (stale doc mention of Taskfile.cloudfront.yaml remains in kind-crossplane/README.md)
…rom config NEEDS LIVE-INSTALL VALIDATION. (companion to the shared CF taskfile deletion in 398f5f9) - drop the cloudfront: include + cloudfront:setup-exposure/sync-domain calls - source ingress_domain_name from config .domain; derive exposure_mode from the new insecure flag; align cert to top-level .certificateArn - seed metadata carries insecure + certificate_arn Verified only: yq parses, task lists 47 kind-kro-ack tasks. NOT install-tested.
.local/ (chat/session scratch artifacts) was accidentally committed via a broad git add. Untrack it from the branch tip and gitignore it. Local files are preserved.
NEEDS LIVE-INSTALL VALIDATION. A generic provider must never shell into a named container. Removed the `docker exec gitlab-gitlab-1 gitlab-rails runner ... ProtectedBranch` branch-unprotect from spokes:enable-crossplane/enable-kro in both providers (4 sites). Branch protection on the fleet/gitops repo is the consumer's responsibility (the workshop configures GitLab when it provisions it). Parse-verified only (yq + task --list). NOT install-tested.
NEEDS LIVE-INSTALL VALIDATION. (b) consumer owns spokes (OAP model): remove spoke enablement + fleet-overlay wiring from the generic platform install in both providers — drop hub:set-overlay-repo, install:phase2-parallel, and the post-install fleet-config pull. Platform install provisions the HUB only + exposes the PlatformCluster abstraction; consumers declare spokes. Also rename the opaque install phases: - install:phase1-parallel -> install:post-sync (parallel work after the first ArgoCD sync: restart identity pods, prefetch, background observability seed) - install:phase1b-parallel -> install:finalize (closing step: restart Langfuse + print URLs) spokes:enable-*/disable-*/set-overlay-repo defs are now unreferenced and move (generic-ized) to workshop/ next. Parse-verified only. NOT install-tested.
NEEDS LIVE-INSTALL VALIDATION. Remove the now-unreferenced, GitLab/fleet-config-coupled task definitions from both generic providers (~1400 lines total): - spokes:enable-crossplane / enable-kro - spokes:disable-crossplane / disable-kro / disable-all - hub:set-overlay-repo - install:phase2-parallel / phase2-spoke-dev / phase2-spoke-prod Kept spokes:seed-provider-identity (bootstrap Crossplane provider IAM — platform mechanism, not GitLab-coupled). Also dropped the spokes:disable-all call from the destroy flow (spoke teardown is now a consumer responsibility, done before platform destroy). The generic providers no longer know about GitLab, fleet-config layout, overlay wiring, or 'tenant: workshop'. Spoke declaration is consumer-owned (OAP model); generic-ized equivalents move to workshop/ next. Parse-verified only (yq + task --list; no dangling task calls; only spokes:seed-provider-identity remains). NOT install-tested.
NEEDS LIVE-INSTALL VALIDATION. Counterpart to removing spoke enablement from the generic providers. Move the GitLab/fleet-config-coupled tasks into the workshop consumer (where GitLab coupling is legitimate — the workshop owns GitLab): - workshop:set-overlay-repo (was hub:set-overlay-repo) - workshop:spokes:enable-crossplane / enable-kro / disable-* / disable-all workshop:install now, after the platform hub install, wires the fleet overlay and declares spoke-dev (crossplane) + spoke-prod (kro) into the GitLab fleet-config — the platform's guarded ApplicationSets reconcile them. Added DOMAIN + K8S_VERSION workshop globals needed by the moved tasks. Parse-verified only (yq + task --list; workshop:spokes:* + set-overlay-repo resolve). NOT install-tested.
… model - config.schema.json: remove the cloudfront.* block (platform no longer provisions CloudFront; gitlabDomain is a workshop concern) and generalize the adminRoleName description (drop the WSParticipantRole example). - common/Taskfile.ray.yaml: prestage-models now takes MODEL_REPO_ID + MODEL_NAME (default TinyLlama for back-compat) instead of hardcoding TinyLlama — keeps Ray generic; the workshop passes the specific model. Parse-verified only (json + yq + task --list). NOT install-tested.
…al clone)
NEEDS LIVE-INSTALL VALIDATION.
Implements the Q-D consumption model: the workshop treats the platform as an
external artifact and invokes its own `task install` as a black box.
- platform.ref unset -> platform IS this repo (workshop/ is a subfolder); used
in place at ROOT_DIR, no clone.
- platform.ref set -> clone platform.repo@ref into .platform/ (pinned
artifact), copy the workshop config.local.yaml (a superset of the platform
contract) into it.
- workshop:install now runs 'cd {{.PLATFORM_DIR}} && task install' instead of
the in-process task-include, so the platform is a true black box.
- added platform:prepare task + PLATFORM_REF/REPO/DIR globals; gitignore .platform/.
Note: ray/spokes still use provider task-includes (fine for the in-place default;
a pinned-clone consumer would get the parent's ray tasks — acceptable, they only
create AWS resources). Parse-verified only. NOT install-tested.
… insecure model The README documented the removed cloudfront exposure mode + deleted Taskfile.cloudfront.yaml (pre-created ALB/NLB, gitlab-distribution, EXPOSURE_MODE). Replace with the current model: consumer provides domain + TLS; `insecure` picks HTTP vs HTTPS+ACM; the LBC owns the ALB normally; CloudFront is a consumer concern.
GitLab is a consumer concern; the generic platform shouldn't advertise a GitLab URL. The workshop's setup-env prints GitLab/platform URLs for the IDE. NOTE: gitlab_domain_name + git_token remain in the cluster-secret seed metadata intentionally — they're a consumed contract for the addon layer (git_token = the gitops-repo credential read by argocd-repo-creds/flux/backstage/cicd; gitlab_domain_name read by backstage/argo-workflows/keycloak GitLab integration). Neutralizing those names + sourcing git_token from config is a separate addon-level effort (tracked). Parse-verified only. NOT install-tested.
…+kubevela The workshop needs two app-deployment spokes, each running Crossplane (app infra) + KubeVela (OAM app delivery). Provision BOTH spoke-dev and spoke-prod via Crossplane so each inherits its environment's enabled-addons overlay (dev/prod both set crossplane: true + kubevela: true) — a single, reliable addon-enablement path. Previously spoke-prod was KRO-provisioned, which does NOT set enable_crossplane/ enable_kubevela in its kro-clusters values, so it would not have gotten those addons. (KRO-provisioning demo dropped; can be restored with explicit enable_crossplane/enable_kubevela if a KRO module is wanted.) Parse-verified only. NOT install-tested.
…om repo root
NEEDS LIVE-INSTALL VALIDATION.
The root Taskfile no longer knows about the workshop:
- removed the workshop: include from the root Taskfile (root is generic-only;
'task --list' at root shows 0 workshop: tasks).
The workshop is now self-contained and run FROM its own folder:
cd workshop && task install
- introduced REPO (= parent of workshop = platform repo root, via ROOT_DIR/..);
all platform paths (config.local.yaml, private/, platform/, .platform) hang
off REPO. Replaced 29 {{.ROOT_DIR}} refs with {{.REPO}}.
- the 3 root-namespace ray refs (:{{.PROVIDER}}:ray:*) become black-box
subprocess calls: 'cd {{.PLATFORM_DIR}} && task {{.PROVIDER}}:ray:*' — matching
how install already invokes the platform. The workshop no longer depends on
being aggregated by the root Taskfile.
Verified: root has no workshop refs; workshop parses + 'task install' resolves
standalone; REPO resolves to the repo root (probe). NOT install-tested.
Remaining root-level workshop item: create-config.sh (workshop config generator,
externally called by CDK) — move needs cross-repo coordination.
… root) NEEDS CDK/SSM REFERENCE UPDATE (external). create-config.sh is the workshop config generator (peeks/WSParticipantRole/ gitlabDomain/feature-branch defaults) — workshop-specific, so it leaves the repo root: - git mv create-config.sh -> workshop/create-config.sh - it still targets the repo-root config.local.yaml (read by the workshop + the in-place platform): OUTPUT_FILE defaults to $REPO_ROOT/config.local.yaml (SCRIPT_DIR/..), and private/ reads use $REPO_ROOT. - updated root README references to workshop/create-config.sh. EXTERNAL FOLLOW-UP: the CDK/SSM SetupIDE bootstrap in the companion repo calls ./create-config.sh at the repo root — it must be updated to workshop/create-config.sh. Syntax-checked (bash -n). NOT run end-to-end.
…om workshop/ Companion to the create-config.sh move: OUTPUT_FILE defaults to $REPO_ROOT/config.local.yaml (SCRIPT_DIR/..) and private/ reads use $REPO_ROOT, so the workshop-located script still generates the repo-root config that the workshop + in-place platform read. Root README references updated to workshop/create-config.sh. (The prior commit recorded only the bare rename because the git add aborted on the stale root pathspec.)
The platform now derives exposure_mode from .insecure (not from an empty domain). CloudFront-mode config left domain:"" but never set insecure, so providers defaulted to domain/HTTPS+ACM mode -> broken ingress. Emit insecure:true so the ALB serves plain HTTP behind CloudFront.
…rom-platform refactor: decouple workshop/CloudFront from the generic platform (WIP)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves GitLab CE out of the EKS hub cluster onto the IDE EC2 instance via docker-compose, exposed through a CDK-managed NLB + CloudFront distribution.
Changes
gitlabArgoCD addon (enabled-addons, hub-config, platform.yaml)gitlab-nlb+gitlab-distributionfromTaskfile.cloudfront.yaml(CloudFront now CDK-managed in platform-engineering-on-eks)gitlab:init-ec2task: wait for GitLab CE readiness, create root token,user1, repos via GitLab API (replaceskubectl execinto the k8s pod)clone-reposwith CloudFront readiness pollGITLAB_DOMAIN_INTnow uses EC2 private IP (private/gitlab-ec2-private-ip) for in-cluster ArgoCD git accessgit_tokenfromseed-secrets(seeded by CDK deploy-time Lambda)Companion MR
platform-engineering-on-eks:
feat/pr-709-kind-crossplane-gitlab-on-ec2→feat/pr-709-kind-crossplaneCloses #754